A data processing system may be arranged in a variety of ways. In some data processing systems, a data processing task is divided into a number of subtasks, and an individual service is implemented to perform each subtask. Each service in the resulting collection of services may act as a consumer and/or a producer of various types of data. Interconnections between the services may be facilitated through the use of a queue structure called a journal. The journal acts as a queue that accepts data in the form of journal entries from one or more producer services (“producer”) and provides the journal entries to a corresponding consumer service (“consumer”). In some data processing systems, the journals and the services are configured to form an end-to-end processing pipeline.
Variations in the processing capability and reliability of individual services in the pipeline may impose limitations on the operation of the pipeline overall. When building a processing pipeline, system administrators attempt to assign an appropriate producer and an appropriate consumer to each journal in the pipeline. However, as the number of alternative producers and consumers increases, and the number of journals in the pipeline increases, a large number of pipeline configurations are possible. Therefore, the efficient management and assignment of consumer/producer resources in a pipelined environment is particularly difficult.